Extreme point characterization of constrained nonstationary infinite-horizon Markov decision processes with finite state space

We study infinite-horizon nonstationary Markov decision processes with discounted cost criterion, finite state space, and side constraints. This problem can equivalently be formulated as a countably infinite linear program (CILP), a linear program with countably infinite number of variables and constraints. We provide a complete algebraic characterization of extreme points of the CILP formulation and illustrate the characterization for special cases. The existence of a K-randomized optimal policy for a problem with K side constraints also follows from this characterization.